Contents

• DaCHS guide
  • Installation
  • DaCHS on Disk
  • Resource descriptors (RDs)
    • Global Metadata
    • Defining Tables
    • Column Definitions
    • Parsing Input Data
      • Sources
      • Grammars
      • Mapping Data
    • Service Definitions
    • Regression Tests
  • Referencing in DaCHS
  • Mixin Introduction
  • Metaprogramming: Macros and LOOPs
• Quick start with DaCHS (dockerised)
  • Publishing Anything Through Obscore
    • Obscore Derived from Typed Service Tables
    • ObsCore Notes
      • Dataset Identifiers
      • target_class
      • s_region
    • Pure Obscore Tables
  • Publishing DaCHS-Managed Tables via TAP
  • Publishing Externally Managed Tables via TAP

DaCHS guide

tl;dr of http://docs.g-vo.org/DaCHS/tutorial.html

Uses example from http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/arihip/q.rd

Installation

• Don't use docker, go native

DaCHS on Disk

• root directory is /var/gavo
• inputs at /var/gavo/inputs
• config at /var/gavo/etc
• logs at /var/gavo/logs
• web interface at /var/gavo/web/

Resource descriptors (RDs)

• XML
• Service consists of RD + data

Global Metadata

• Root element <resource> should define database schema, e.g. <resource schema="arihip">

• One RD per schema

• Example:

    <meta name="title">ARIHIP astrometric catalogue</meta>
    <meta name="creationDate">2010-11-03T10:13:00</meta>
    <meta name="description">
      The catalogue ARIHIP has been constructed by
      selecting the 'best data' for a given star from combinations of HIPPARCOS
      data with Boss' GC and/or the Tycho-2 catalogue as well as the FK6.  It
      provides 'best data' for 90 842 stars with a typical mean error of
      0.89 mas/year (about a factor of 1.3 better than Hipparcos for this
      sample of stars).
    </meta>
    <meta name="creator">Wielen, R.; Schwan, H.; Dettbarn, C.; et al</meta>
  
    <meta name="subject">Catalogs</meta>
    <meta name="subject">Astrometry</meta>
    <meta name="subject">Stars: Proper Motions</meta>
    <meta name="type">Catalog</meta>
  
    <meta name="coverage.waveband">Optical</meta>
  
    <coverage>
      <spatial>0/0-11</spatial>
      <spectral>2.721e-19 4.138e-19</spectral>
      <temporal>1989-09-01 1993-08-15</temporal>
    </coverage>
  
    <FEED source="//procs#license-cc-by" what="ARIHIP"/>
  
    <meta name="_longdoc" format="rst">
      The ARIHIP Catalogue is a suitable combination of the results of the
      HIPPARCOS astrometry satellite with ground-based data.
  
      [...]
    </meta>
  
    <meta name="source">2001VeARI..40....1W</meta>
  
    <meta name="_intro" format="rst"> <![CDATA[
      For advanced queries on this catalogue use ADQL_
      possibly via TAP_
  
      .. _ADQL: /adql
      .. _TAP: /tap
    ]]> </meta>

  • meta is global metadata for RD
  • _ indicates reserved for DaCHS internal use
  • capitalised attributes are active tags. FEED specifies values are to be injected from elsewhere
  • some special elements, e.g. coverage

• metadata is inherited, e.g. if service in an RD has no title, it will use RD title

Defining Tables

• A resource may contain multiple table definitions
• Example:

    <table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex" primary="hipno">

Column Definitions

• For every column in the table there will be a column element with attributes that are a mixture of VO-specific (e.g. unified content descriptors or ucd) and SQL (e.g. name or type) e.g.

    <column name="hipno" type="integer" ucd="meta.id;meta.main"
      tablehead="HIP id" verbLevel="1"
      description="Number of the star in the HIPPARCOS Catalogue (ESA 1997)."
      required="True"/>
    <column name="srcSel" type="text" ucd="meta.flag"
      tablehead="Source" verbLevel="25"
      description="Source of the astrometric solution"
      note="src"/>

• Can specify a child element of column, values. This allows one to set min/max, enumerate elements, specify null values... e.g.

    <column name="n_obs" type="integer"
        description="Number of
        observations, NULL if interpolated data">
      <values nullLiteral="-1"/>
    </column>

• Can also specify an index to be created for a table using a child element of table, index, e.g.

    <index columns="mv"/>

Parsing Input Data

• Define how data fills the above tables
• Example:

    <data id="import">
      <sources>data/data.txt.gz</sources>

Sources

sources child element defines path to source data

Grammars

• "Grammars" turn an input into a sequence of dictionaries, output is a rawdict (normally string -> string mapping)
• The rowmaker turns a rawdict into rowdicts, which is a mapping of rawdict column to a column that can be understood by the database
• Example:

    <columnGrammar topIgnoredLines="9" preFilter="zcat">
      <colDefs>
        hipno:     3-8
        srcSel:    47-49
        alphaHMS:  59-73
        [...]
      </colDefs>

• Can tell DaCHS to inspect how a grammar is acting on input by e.g. dachs imp -M 100 --dumpRows q.rd | less
• Can preprocess files by specification of attributes e.g. topIgnoredLines, preFilter. The latter allows input to be preprocessed by a shell command
• Can separate fields by different methods using different element names, e.g. reGrammar for delimiter or columnGrammer for fixed-width

Mapping Data

• Make element brings together a table with a recipe on how to fill it (row maker)

• Example:

    <make table="main">
      <rowmaker idmaps="*">
        <var name="raj2000">hmsToDeg(@alphaHMS, None)</var>
        <var name="dej2000">dmsToDeg(@deltaDMS, None)</var>
        ...
        <map dest="kbin">parseWithNull(@kbin, str, "9")</map>
        <map dest="vrad">parseWithNull(
          @vrad, lambda a:float(killBlanks(a)), "")</map>

  • a rowmaker element has three kinds of children
    • var: assignment of an expression value in the rawdict
    • map: mapping of python expression to value in the destination rowdict
    • apply: manipulation of rawdicts and rowdicts in python code
  • the @identifier syntax expands to whatever the rawdict has for the key identifier

• In most cases map will suffice if no other manipulation is required and a default conversion (e.g. python int to sql integer) is OK

• Example:

    <rowmaker>
      <map dest="evi" src="evi"/>
      <map dest="av" src="av"/>
      <map dest="ai" src="ai"/>
    </rowmaker>

  or shorthand:

    <rowmaker simplemaps="evi:evi,av:av,ai:ai"/>

  or further shorthand where column names and identifiers have the same name:

    <rowmaker idmaps="evi,av,ai"/>

  or even shorter:

    <rowmaker idmaps="*"/>

Service Definitions

• Service Definitions describe how to get data out of the database.

• To understand this, need to understand basic DaCHS architecture:

  • cores: the database query
  • renderers: a way of interpreting and presenting data from/to the client
  • "services": a binding of these two

• Example:

    <service id="cone" allowed="scs.xml,form">
      <meta name="shortName">arihip cone</meta>
      <meta name="testQuery">
        <meta name="ra">9.4076</meta>
        <meta name="dec">9.6414</meta>
        <meta name="sr">1.0</meta>
      </meta>
  
      <dbCore queriedTable="main">
        <FEED source="//scs#coreDescs"/>
        <condDesc buildFrom="mv"/>
        <condDesc>
          <inputKey original="hipno" required="False"/>
        </condDesc>
      </dbCore>
  
      <publish render="scs.xml" sets="ivo_managed"/>
      <publish render="form" sets="ivo_managed,local"/>
      <outputTable verbLevel="20"/>
    </service>

  • allowed attribute of service shows the renderers available, i.e. xml and form
  • cores are specified as dbCore elements, i.e. here it is the in-built scsCore. Other services include a generic dbCore, nullCore (where no server-side computation is required), datalinkCore as well as custom written cores
  • Each dbCore element generates a single table query against queriedTable using condition descriptors condDesc
  • The child attributes original and buildFrom are the most commonly used attributes of condDesc:
    • original specifies a key that will be input by the user, e.g. by a form
    • buildFrom builds an input definition from the column defined, e.g. mv in this example
  • See http://docs.g-vo.org/DaCHS/ref.html#scs-coredescs

• They can be declared at a RD level if shared by many services, in which case specify <dbCore core='id.of.element'>

Regression Tests

• Use regTest child element of regSuite. Basically consists test case with known result, e.g.

    <regSuite title="ARIHIP regression">
      <regTest title="ARIHIP form service appears to work.">
        <url parSet="form" hscs_sr="1.0" hscs_pos="0.00 1.08"
          >cone/form</url>
        <code>
          self.assertHasStrings("PM RA (STP)",
            "1.0890086361", "-5.00", "Note bin")
        </code>
      </regTest

Referencing in DaCHS

• RDs can be referenced relative to the inputs directory
• When referencing particular elements, use XML style syntax e.g. original="main" references the main element in the current RD
• Can reference tables with children using a ., e.g. myres/q#main.ra references column ra in the main table in myres/q
• // references in-built RDs.

Mixin Introduction

• Mixins are used to make sure that tables meet IVOA standards, e.g. contain the right columns, metadata, indices
• Mixins are essentially in-built (//) RDs

Metaprogramming: Macros and LOOPs

...skipped for now.

Quick start with DaCHS (dockerised)

Start DaCHS via docker (contains both postgres/dachs in one container):

eng@ubuntu:~$ docker run -it -p 8080:8080 --name dachs gavodachs/dachs
root@a27bb87ffb6e:/#service postgresql start
root@a27bb87ffb6e:/#dachs serve debug

Navigate to localhost at port 8080

Add data and set authority:

eng@ubuntu:~$ docker exec -it dachs /bin/bash
root@a27bb87ffb6e:/var/gavo/inputs# mkdir -p arihip/data
root@a27bb87ffb6e:/var/gavo/inputs# curl http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/arihip/q.rd -o arihip/q.rd
root@a27bb87ffb6e:/var/gavo/inputs# curl http://dc.g-vo.org/arihip/q/cone/static/data.txt.gz -o arihip/data/data.txt.gz
root@a27bb87ffb6e:/var/gavo/inputs# gavo imp arihip/q && gavo pub arihip/q
root@a27bb87ffb6e:/var/gavo/inputs# vim /etc/gavo.rc

[web]
sitename: Dachs-on-Docker
bindAddress:
serverPort: 8080
serverURL: http://localhost:8080
logFormat: combined

[ivoa]
authority: test.rmb

Reset the DaCHS server:

root@a27bb87ffb6e:/# ^C
root@a27bb87ffb6e:/#dachs serve debug

Import the RD and publish:

root@a27bb87ffb6e:/var/gavo/inputs# gavo imp arihip/q && gavo pub arihip/q

Reset the DaCHS server:

root@a27bb87ffb6e:/# ^C
root@a27bb87ffb6e:/#dachs serve debug

Navigate to localhost at port 8080 again

Use some target in SCS e.g. HD 27846

Publishing Anything Through Obscore

• Supersedes some previous services e.g. SIAP, SSAP
• In DaCHS, the obscore table is a view, mapping resources from other individual tables on to this schema
  • These mappings are created using a mixin from the //obscore RD with filled out parameters that are used to construct the required SQL data defininition language (DDL) statements that comprise the view

Obscore Derived from Typed Service Tables

• If there is already a typed service, e.g. SIAP (simple image access protocl), it is possible to use something like mixin="//obscore#publishSIAP" to add existing data
  • Adding metadata can be done by e.g.<mixin sResolution="0.5" calibLevel="2">//obscore#publishSIAP</mixin>

ObsCore Notes

Dataset Identifiers

• obs_publisher_did column, often used as primary key (i.e. unique)

target_class

...skipped for now.

s_region

...skipped for now.

Pure Obscore Tables

...skipped for now.

Publishing DaCHS-Managed Tables via TAP

• To publish a table through TAP add adql=True to table definition, add <publish/> to table body and update metadata dachs imp -m
• To publish this table at root page of site add <publish sets="ivo_managed, local"/>

Publishing Externally Managed Tables via TAP

• Discouraged, but necessary if data lives in another table. Example:

    <resource schema="mydata">
      <meta name="title">My great table</meta>
      <meta name="creationDate">... (more metadata)
  
      <table id="values" onDisk="True" adql="True">
         <column name="id" type="bigint" unit="" ucd="meta.id;meta.main">
           <description>id of object covered here</description></column>
      </table>
  
      <data id="d" updating="True">
        <publish/>
        <make table="values"/>
      </data>
    </resource>

  • <data> element has no sources
  • updating makes sure that DaCHS doesn't tear down the table
  • use -m flag during DaCHS import to import only metadata

• DaCHS must be able to modify these tables, so ensure that privileges are granted to gavoadmin in the database